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ABSTRACT 


This study is part of a research programme investigating the dynamics and impacts of learning engagement in a 
challenge-based digital learning environment. Learning engagement is a multidimensional concept which includes an 
individual’s ability to behaviourally, cognitively, emotionally, and motivationally engage in an on-going learning 
process. Challenge-based learning gives significant freedom to the learner to decide what and when to engage and 
interact with digital learning materials. In light of previous empirical findings, we expect that learning engagement is 
positively related to learning performance in a challenge-based digital learning environment. This study was based on 
data from the Curtin Challenge platform, including transaction data of 13,655 students. Findings indicate that attributes of 
learning engagement in challenge-based digital learning environments are, as expected, positively related to learning 
performance. Implications point toward the need for personalised and adaptive learning environments to be developed in 
order to cater for the individual needs of learners in challenge-based digital learning environments. 
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1. INTRODUCTION 


Learners differ in their reasons for engaging in learning tasks and these inter-individual differences require 
personalised support while learning (Schunk & Zimmerman, 1994). In addition, research also reports 
intra-individual differences in engagement, i.e., during a learning dependent-progression engagement 
changes over time and requires adaptive support to cater for the learners’ needs (Ifenthaler & Seel, 2005). 

Learning engagement is generally regarded as the time and effort an individual invests on a specific 
learning activity (Kuh, 2009). Several studies focussing on learning engagement support the assumption that 
higher engagement of a learner corresponds with higher learning outcomes (Carini, 2012). However, most of 
these studies have been conducted in face-to-face learning environments. Accordingly, a confirmation of 
these findings in digital learning environments is still lacking. 

This study seeks to close this gap by investigating the dynamics of learning engagement in a 
challenge-based digital learning environment using a data analytics approach. The context of the presented 
study is set in the Curtin Challenge which is a mobile ready interactive learning delivery platform that 
illustrates several features of game-inspired challenge-based learning while adding a layer of big data 
collection to enable research into teaching and learning. A learner interacts with Curtin Challenge content by 
pointing, clicking, sliding items, vocalizing, taking pictures and drawing as well as watching, listening, 
reading and writing as in typical digital learning environments. 


2. LEARNING ENGAGEMENT 


Learning engagement is a multidimensional concept and understood as the individual’s ability to 
behaviourally, cognitively, emotionally, and motivationally interact with learning artefacts in an on-going 
learning process (Wolters & Taylor, 2012). A generally accepted assumption is that the more students engage 
with a subject matter or phenomenon in question, the more they tend to learn (Carini, Kuh, & Klein, 2006). 
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This assumption is consistent with the theory of self-regulated learning (Zimmerman, 2002) and concepts of 
engagement (Fredricks & McColskey, 2012). Accordingly, learning engagement is positively linked to 
desirable learning outcomes or learning performance (Klein, Kuh, Chun, Hamilton, & Shavelson, 2005). 

While learning performance is linked closely with behaviours (Bandura, 1993), several assumptions are 
associated to the relationship between the performance of an individual and learning engagement. For 
example, Chen (2017) investigated the relationship between learning engagement and learning performance 
of students of ten schools based in Taiwan. Findings of the multilevel analysis indicate a significant positive 
relationship between learning engagement and learning performance. Recent findings also document that 
serious games drive learning engagement (Peng, Cao, & Timalsena, 2017). Similar implications focussing on 
learning engagement and learning performance have been reported in other contexts (Flowerday & Shell, 
2015; Lin et al., 2016; Pourbarkhordari, Zhou, & Pourkarimi, 2016). 

An impressive number of research studies have been conducted in the field of cognitive load with links to 
task characteristics and learning engagement (Kirschner, Kester, & Corbalan, 2011; van Merriénboer & 
Sweller, 2005). This line of research assumes an active role of the learner in learning processes, i.e., learners 
select tasks relevant to them (Corbalan, Kester, & van Merriénboer, 2011) and are actively engaged while 
interacting with the learning environment (Schwamborn, Thillmann, Opfermann, & Leutner, 2011). 

In addition, research on reading utilises reading time measurements in order to identify learning 
engagement and linking those to learning performance (Graesser, Millis, & Zwaan, 1997). The general 
assumption is that the intensity of mental effort aimed at achieving a greater understanding, i.e., time spent 
on reading task, is critical during learning. Findings indicate that increased reading times as a sign of greater 
learning engagement are positively related to learning performance measured as comprehension scores 
(Miller, 2015; Miller et al., 2014). 


3. CHALLENGE-BASED LEARNING ENVIRONMENT 


The Curtin Challenge digital learning platform (http://challenge.curtin.edu.au) supports individual and 
team-based learning via gamified, challenge-based, open-ended, inquiry-based learning experiences that 
integrate automated feedback and rubric-driven assessment capabilities. 


Curtin Careers Curtin Leaders Program Curtin UniReady 
Careers Illuminate Leadership Challenge English Challenge 


Figure 1. Three challenges currently available 


A challenge is regarded as a collection of information and corresponding tasks linked to specific learning 
outcomes. Currently, there are three Challenges offered by Curtin University: Careers Illuminate, Leadership 
Challenge, and English Challenge (see Figure 1). This study includes analysis from the Career and 
Leadership Challenges which both require approximately up to one hour of learning time. Career Challenge 
includes 14 modules while Leadership Challenge includes eleven modules (see Figure 2 for individual 
modules). 
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Careers Illuminate VG Leadership Challenge 


‘Challenge Hub Overview Modules «Challenge Hub Overview Modules 


Figure 2. Modules overview for Career and Leadership Challenges 


The design features of each module contain up to five activities including one to three different learner 
interactions or tasks. For example, the module Who am I in the Career Challenge is a collection of five 
activities containing learning interactions, such as choosing from among options, writing a short response to 
a prompt, spinning a wheel to create random prompts, creating, organising, and listing ideas, or matching 
items. Each page can contain one or several such interactions, and the learner does not have to submit the 
page in order for the data to be captured. Data is constantly being captured, which creates information about 
the timing, sequence, and completeness as well as the content of the interactions (i.e., navigation event and 
sequences). The data record is thus highly granular, providing an opportunity to examine the dynamics of the 
activity as well as the contents of the artefacts created by the learner for every click on every activity or 
module page. 


4. HYPOTHESES OF THE PRESENT STUDY 


In light of previous empirical findings on learning engagement (Chen, 2017; Flowerday & Shell, 2015; 
Kirschner et al., 2011; Miller, 2015; Miller et al., 2014), we expect that learning engagement is positively 
related to learning performance in a challenge-based digital learning environment. Attributes of learning 
engagement in the challenge-based digital learning environment are conceptualised through several actions: 
(a) launching a specific activity (task), (b) spending active time on the task, (c) entering a written response, 
and (d) finishing a task. The learning performance measured in this study is computed by the number of 
correct answers in a subset of tasks designed with embedded feedback to the student. The hypotheses of this 
study focus on the attributes of learning engagement and its relation to learning performance in both Career 
and Leadership Challenge. We assume that launching specific activities (tasks) is related to the learning 
performance in challenge-based digital learning environments (Hypothesis 1). Further, we assume that 
spending active time on tasks is related to the learning performance (Hypotheses 2). Also, we expect that the 
length of written responses is related to the learning performance (Hypothesis 3). The final assumption 
focusses on the relationship between finishing tasks and learning performance (Hypothesis 4). 


5. METHOD 


5.1 Data Source 


The data set of the Career Challenge consists of 52,675,225 rows of raw data containing information of 
Nc = 8,951 students (3,571 male; 5,380 female) with an average age of M = 25.72 years (SD = 6.64). The 
Leadership Challenge includes data from N, = 4,704 students (1,825 male; 2,879 female) with an average age 
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of M = 23.96 years (SD = 5.47) with information stored in 19,517,647 rows of raw data. In a period of 24 
months (January 2016 — January 2018), students spent a total of 10,239 hours interacting with the Career 
Challenge and 14,546 hours interacting with the Leadership Challenge. 


5.2 Data Analytics Strategy 


Raw data from the Career and Leadership Challenge were cleaned and transformed into a transaction data set 
in which each row represents an event of one user. The dependent variable learning_performance (LP) was 
computed as the number of correct answers in an activity. The variables reflecting attributes of learning 
engagement were computed as follows: launching_task (LT) as the number of activities started by a student; 
time_on_task (TT) as the duration in seconds spent in an activity; written_response (WR) as the number of 
words submitted by a student; finishing_task (FT) as the number of activities finished by a student. 


6. RESULTS 


In order to test the above presented four hypotheses, regression analyses were computed to determine 
whether attributes of learning engagement (i.e., launching task, time on task, written response, finishing task) 
were significant predictors of learning performance in challenge-based digital learning environments. The 
analyses were computed separately for the Career and Leadership Challenge. 


6.1 Career Challenge 


Table 1 shows zero-order correlations of attributes of learning engagement and learning performance for the 
Career Challenge. All correlations were significant at p < .001. High positive correlations were found 
between launching task (LT; M = 6.73; SD = 8.95) and learning outcome (LP; M = 8.38; SD = 13.19), time 
on task (TT; M = 4118.09; SD = 6623.88), as well as written response (WR; M = 166.92; SD = 284.62). 
Moderate positive correlations were found for written response and learning outcome as well as time on task. 
Low positive correlations were found for the remaining variable combinations. 


Table 1. Zero-order correlations, means and standard deviations of attributes of learning engagement and learning 
performance for the Career Challenge 


Zero-Order r 


LT TT WR FT LP 
LT - 
TT TT LEE - 
WR TIA BRE OBS HHH = 
FT BOOT DO0EEE. 3S LEE - 
LP B39F* — O2B*HK — -OOO*F* 340% FE - 
M 6.73 4118.09 166.92 1.24 8.38 
SD 8.95 6623.88 284.62 4.40 13.19 


Note. *** p < .001; LP = learning outcome; LT = launching task; 
TT = time on task; WR = written response; FT = finishing task; 
Nc = 8,951 


The linear regression analysis for the Career Challenge is presented in Table 2, yielding a AR’ of .713 
(F(4, 8950) = 5568.79, p < .001). Clearly, the number of activities started by a student (LT; 2 = .80, p < .001) 
positively predicted the learning performance. In addition, the number of activities finished by a student (FT; 
f = .04, p < .001) and the number of words submitted by a student (WR; f = .13, p < .001) positively 
predicted the learning performance. In contrast, the duration students spent on a task (TT; 6 = -.09, p < .001) 
was negatively correlated with the learning performance. 
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Table 2. Regression analyses predicting learning performance by attributes of learning engagement for the Career 


Challenge 
R AR’ B SEB B 
LP 713 713 
LT tg7.; «- 3015 .80*** 
TT .001 .001 -. 09% 
FT 115 018 04" 
WR .006 001 13*#* 


Note. *** p < .001; LP = learning performance; 
LT = launching task; TT = time on task; 
FT = finishing task; WR = written response; Nc = 8,951 


In sum, the four hypotheses are accepted for the Career Challenge, confirming significant relationships 
between attributes of learning engagement and learning performance. 


6.2 Leadership Challenge 


Table 3 shows zero-order correlations of attributes of learning engagement and learning performance for the 
Leadership Challenge. 


Table 3. Zero-order correlations, means and standard deviations of attributes of learning engagement and learning 
performance for the Leadership Challenge 


Zero-Order r 


LT TT WR FT LP 
LT - 
TT 698% : 
WR LOGTEES 1 SOFTEE : 
FT 1.00*** = 697*** 75 Q% 2 
LP OLE — 667*** TILER ODT - 
M 26.74 11132.30 661.78 26.52 10.76 
SD 22.97 14535.29 782.97 23.05 11.96 


Note. *** p < .001; LP = learning outcome; LT = launching task; 
TT = time on task; WR = written response; FT = finishing task; N; = 4,704 


The linear regression analysis for the Leadership Challenge is presented in Table 4, yielding a AR’ of .850 
(F(4, 4703) = 6652.32, p < .001). 

The number of activities started by a student (LT; 8 = 1.50, p < .001) positively predicted the learning 
performance. In addition, the duration students spent on a task (TT; 6 = .05, p < .001) positively predicted the 
learning performance. In contrast, the number of activities finished by a student (FT; £ = -.61, p < .05) was 
negatively correlated with the learning performance. 

In sum, the hypotheses 1, 2 and 4 are accepted for the Leadership Challenge, confirming significant 
relationships between attributes of learning engagement and learning performance. 


Table 4. Regression analyses predicting learning performance by attributes of learning engagement for the Leadership 


Challenge 
R AR’ B SEB B 
LP 850 850 
164 .782 127 1.50% 
TT .038 .000 O5*#* 
FT -.318 129 -.61* 
WR 001 .000 .00 


Note. * p < .05; *** p < .001; LP = learning performance; LT = launching task; 
TT = time on task; FT = finishing task; WR = written response; N; = 4,704 
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7. DISCUSSION 


The times of direct comparisons of technology-mediated and face-to-face learning environments are over 
(Alavi & Leidner, 2001), hence, research needs to identify key factors influencing learning processes and 
learning outcomes. This study aimed to investigate the dynamics of engagement in challenge-based digital 
learning environments and its relationship to learning performance. Hypotheses were developed based on 
previous research from face-to-face learning environments. Our analyses focussed on data from the 
Challenge platform including transaction data from 13,655 students. 

The analytic results showed that learning engagement in challenge-based online learning environments is 
significantly related to learning performance. These findings support previous studies conducted in 
face-to-face situations (Chen, 2017; Lin et al., 2016; Pourbarkhordari et al., 2016). Significant attributes 
predicting the learning performance of the student appeared to be the number of activities started and the 
number of activities finished by a student. This is a reflection of active engagement with the learning 
environment (Kirschner et al., 2011). At the same time, better learners seem to spend less time on a specific 
task in the Career Challenge. This may be interpreted as a reflection of existing prior knowledge or a 
progression towards an advanced learner (Ifenthaler & Seel, 2005). Another significant indicator predicting 
learning performance in the Career Challenge was the number of words submitted in open text activities. On 
a surface level, these findings are also related to studies conducted in writing research and clearly reflect the 
impact of the variation in learning engagement (Graesser et al., 1997; Miller et al., 2014). 

This study and its findings are limited in several aspects that must be addressed. First, due to limited 
access of student data, for example, course load, past academic performance, or personal characteristics, 
linking additional data to the reported engagement and performance measures has not yet occurred. 
Combining such additional data, we expect will provide a more detailed insight into the multidimensional 
concepts to be investigated in a future study. Second, both challenges did not include an overall performance 
measure which has been validated against an outside criterion. Accordingly, a revision of the learning and 
assessment design should include additional or revised measures which follow accepted criteria or 
competence indicators. However, without the externally validated benchmarks, there is sufficient available 
data which can be used to improve the existing learning design through algorithms focussing on design 
features and navigation sequences of learners (Agrawal, Golshan, & Papalexakis, 2016; Ifenthaler, Gibson, & 
Dobozy, 2018; Lockyer, Heathcote, & Dawson, 2013). Third, as we included the analysis of open text 
answers in our analysis model, this approach is limited by the overall potential of the simple approach natural 
language processing (NLP). Further development of our analysis in future studies will include a focus on 
deeper levels of syntactic complexity, lexical sophistication, and quality of writing as well as a deep semantic 
analysis compared to expert solutions (Crossley, 2013; Ifenthaler, 2014). 


8. IMPLICATIONS AND FUTURE RESEARCH 


Analyses of the learning performance transcript, even when automated and multileveled, is a mixture of 
conditional and inferential interpretation that can utilize several frames of reference while adding layers of 
interpreted evidence, insights concerning the complexity and additional dimensionality to our understanding 
of the performance and our ability to re-present the performance in the light of our understandings (Gibson 
& Ifenthaler, 2018). 

The Curtin Challenge platform is being developed to support both individual and team-based learning in 
primarily open-ended ill-structured problem solving and project-based learning contexts (Eseryel, Law, 
Ifenthaler, Ge, & Miller, 2014). The platform can also support self-guided learning, automated feedback, 
branching story lines, self-organizing teams, and distributed processes of mentoring, learning support and 
assessment (Gibson, 2018; Gibson & Ifenthaler, 2018). 

The data traces captured by the Curtin Challenge platform are highly detailed, with many events per 
learning activity, which brings the potential for measuring indicators of physical, emotional and cognitive 
states of the learner. The data innovation of the Curtin Challenge platform is the ability to capture 
event-based records of higher frequency with the potential to analyse higher dimensional aspects of learning 
engagement, which we believe may be in turn useful for analysis of the embedded learning design’s 
effectiveness and impact on the physical, emotional and cognitive layers of learning caused or influenced by 
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digital engagements. The data from the challenge-based learning platform forms a high-resolution analytics 
base on which researchers can conduct studies into learning design and into how to achieve better outcomes 
in scalable digital learning experiences (Gibson, 2018; Gibson & Jackl, 2015). 

Future research will focus on the analysis of several large extant data sets from the Curtin Challenge 
platform. Currently, the possibility of adaptive algorithms based on learning engagement and learning 
performance are being investigated. Such algorithms will enable meaningful micro analysis of individual 
performance as well as personalised and adaptive feedback to the learner whenever it is needed. 
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